Papers with latency-sensitive applications
Pieces of Eight: 8-bit Neural Machine Translation (N18-3)
Copied to clipboard
| Challenge: | Neural machine translation models are trained using 32-bit floating point values and have improved fluency and adequacy. |
| Approach: | They propose to use 8-bit quantization to train models using 32-bit floating point values and show that 8- bit translation makes a non-negligible impact in terms of speed with no degradation in accuracy and adequacy. |
| Outcome: | The proposed method improves accuracy and accuracy without degradation in accuracy and adequacy. |
DART: Distilling Autoregressive Reasoning to Silent Thought (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that use Chain-of-Thought (CoT) have been slow to deploy in real-time applications due to its autoregressive nature. |
| Approach: | They propose a framework that replaces autoregressive CoT with non-autoregressive Silent Thought (ST) the framework uses a lightweight Reasoning Evolvement Module to align hidden states with the CoT pathway and a Reasoning Embedment Module (REM) during inference, only the ST pathway is activated, enabling the ST tokens to evolve into informative embeddings. |
| Outcome: | The proposed framework replaces autoregressive CoT with non-autoregressive Silent Thought (ST) it enables LLMs to generate answers directly from ST tokens without additional computational cost . |
Differentially Private Knowledge Distillation via Synthetic Text Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language models (LLMs) are achieving state-of-the-art performance in many downstream tasks, but data privacy is a major challenge for practitioners. |
| Approach: | They propose a differentially private knowledge distillation algorithm that exploits the knowledge of a teacher LLM and a student's output distribution. |
| Outcome: | The proposed algorithm significantly improves the utility over baselines on the Big Patent dataset, with strong privacy parameters, =2. |